1. Field of the Invention
The present invention relates to an image processing apparatus with interframe interpolation capabilities, and more particularly, to an image processing apparatus which reproduces a smooth sequence of frames by applying an interframe interpolation to video images with some frame dropped.
2. Description of the Related Art
Some video communications systems, such as videoconferencing systems, are designed to use ordinary low-speed telecommunication lines to deliver image information to the receiving ends. In such systems, the frame rate (i.e., the number of frames transmitted in a unit time) is dynamically controlled in accordance with the amount of information to be actually delivered. More specifically, when the source pictures contained a large amount of video information, some frames may be dropped at the sending end to reduce the information transmitted over the communications channel. This frame subsampling operation, however, can degrade smoothness of the pictures reproduced at the receiving ends; it makes rapid movements of on-screen objects appear jerky. The present invention is addressed to the solution of such problems.
Japanese Patent Application Laid-open Publication No. 3-109677 (1991), for example, discloses an image processor with interframe interpolation capabilities. This conventional image processor produces a smooth and continuous succession of video images from time-discrete source pictures sampled at certain time intervals, which feature allows the processor to be used to create animation from a limited number of key-frames or to display images captured by ultrasound imaging devices for medical treatment.
FIG. 9 shows a typical configuration of such a conventional image processor with frame interpolation functions. Specifically, source frame images captured by a camera 101 are stored into a frame memory 102, frame by frame. The frame memory 102 supplies a moment calculation unit 103 with two consecutive frame images captured at discrete points in time, (tn) and (tn+1), as illustrated in FIGS. 10(A) and 10(B). In each frame image provided, the moment calculation unit 103 recognizes a moving object 107 by approximation to an ellipse, and calculates its first-order moment axis 108 and second-order moment axis 109. Next, an affine parameter calculation unit 104 calculates affine parameters (or coefficients for affine transformation) through evaluation of geometrical relation between the first-order and second-order moment axes 108 and 109 at time point (tn) and those at time point (tn+1).
Based on the affine parameters calculated above, a predictive affine parameter calculation unit 105 calculates another set of affine parameters for an intermediate time point (tn+t) between the two time points (tn) and (tn+1); i.e., tn&lt;tn+t&lt;tn+1. The obtained affine parameters are then supplied to a predictive object image generation unit 106. With the affine parameters for time point (tn+t), the predictive object image generation unit 106 conducts an affine transformation to the image of the moving object 107 included in the source frame image at time point (tn) or (tn+1) sent from the frame memory 102, thereby generating an in-between frame image at time point (tn+t).
As described above, the conventional image processor of FIG. 9 calculates the first-order moment axis 108 and second-order moment axis 109 to determine the object's orientation. That algorithm, however, consumes a large amount of computing power to perform two-dimensional sum-of-products operations over the entire image of each moving object. This leads to a difficulty in obtaining frame-interpolated video images in a real-time fashion. As mentioned earlier, such conventional image processors can be used to create animation or to display images captured by ultrasound imaging devices for medical purposes. However, it should be noticed here that these are all off-line, or non-real-time, applications. Conversely, the conventional image processors with interframe interpolation capabilities can not be applied to, for example, videoconferencing systems where real-time image processing is essential.
Meanwhile, the rotation angle of the moving object 107 is one of the affine parameters that must be identified. The conventional image processors calculate this parameter from the first-order moment axis 108 and second-order moment axis 109 at two points in time (tn) and (tn+1). But here lies another problem with the conventional algorithms that the range of rotation angles that can be correctly identified is limited to 90 degrees at most.
More specifically, consider that the first-order moment axis 108 at time point (tn) and the same at time point (tn+1) form an angle .alpha., where 0&lt;.alpha..ltoreq.90 [degrees]. Although this condition can actually happen when the object has rotated by just a degrees or (180-.alpha.) degrees, the conventional image processing algorithms are unable to discriminate whether the true rotation angle was a degrees or (180-.alpha.) degrees because of the symmetrical nature of the elliptic approximation models that they use. In practice, the conventional image processors are designed not to interpret it as (180-.alpha.) degrees, but take .alpha. degrees as the rotation angle in such a situation. That is, they cannot identify the affine parameters correctly when the actual rotation of an object exceeds 90 degrees.